119 research outputs found
Improving Big Data Visual Analytics with Interactive Virtual Reality
For decades, the growth and volume of digital data collection has made it
challenging to digest large volumes of information and extract underlying
structure. Coined 'Big Data', massive amounts of information has quite often
been gathered inconsistently (e.g from many sources, of various forms, at
different rates, etc.). These factors impede the practices of not only
processing data, but also analyzing and displaying it in an efficient manner to
the user. Many efforts have been completed in the data mining and visual
analytics community to create effective ways to further improve analysis and
achieve the knowledge desired for better understanding. Our approach for
improved big data visual analytics is two-fold, focusing on both visualization
and interaction. Given geo-tagged information, we are exploring the benefits of
visualizing datasets in the original geospatial domain by utilizing a virtual
reality platform. After running proven analytics on the data, we intend to
represent the information in a more realistic 3D setting, where analysts can
achieve an enhanced situational awareness and rely on familiar perceptions to
draw in-depth conclusions on the dataset. In addition, developing a
human-computer interface that responds to natural user actions and inputs
creates a more intuitive environment. Tasks can be performed to manipulate the
dataset and allow users to dive deeper upon request, adhering to desired
demands and intentions. Due to the volume and popularity of social media, we
developed a 3D tool visualizing Twitter on MIT's campus for analysis. Utilizing
emerging technologies of today to create a fully immersive tool that promotes
visualization and interaction can help ease the process of understanding and
representing big data.Comment: 6 pages, 8 figures, 2015 IEEE High Performance Extreme Computing
Conference (HPEC '15); corrected typo
Performance Measurements of Supercomputing and Cloud Storage Solutions
Increasing amounts of data from varied sources, particularly in the fields of
machine learning and graph analytics, are causing storage requirements to grow
rapidly. A variety of technologies exist for storing and sharing these data,
ranging from parallel file systems used by supercomputers to distributed block
storage systems found in clouds. Relatively few comparative measurements exist
to inform decisions about which storage systems are best suited for particular
tasks. This work provides these measurements for two of the most popular
storage technologies: Lustre and Amazon S3. Lustre is an open-source, high
performance, parallel file system used by many of the largest supercomputers in
the world. Amazon's Simple Storage Service, or S3, is part of the Amazon Web
Services offering, and offers a scalable, distributed option to store and
retrieve data from anywhere on the Internet. Parallel processing is essential
for achieving high performance on modern storage systems. The performance tests
used span the gamut of parallel I/O scenarios, ranging from single-client,
single-node Amazon S3 and Lustre performance to a large-scale, multi-client
test designed to demonstrate the capabilities of a modern storage appliance
under heavy load. These results show that, when parallel I/O is used correctly
(i.e., many simultaneous read or write processes), full network bandwidth
performance is achievable and ranged from 10 gigabits/s over a 10 GigE S3
connection to 0.35 terabits/s using Lustre on a 1200 port 10 GigE switch. These
results demonstrate that S3 is well-suited to sharing vast quantities of data
over the Internet, while Lustre is well-suited to processing large quantities
of data locally.Comment: 5 pages, 4 figures, to appear in IEEE HPEC 201
Enabling On-Demand Database Computing with MIT SuperCloud Database Management System
The MIT SuperCloud database management system allows for rapid creation and
flexible execution of a variety of the latest scientific databases, including
Apache Accumulo and SciDB. It is designed to permit these databases to run on a
High Performance Computing Cluster (HPCC) platform as seamlessly as any other
HPCC job. It ensures the seamless migration of the databases to the resources
assigned by the HPCC scheduler and centralized storage of the database files
when not running. It also permits snapshotting of databases to allow
researchers to experiment and push the limits of the technology without
concerns for data or productivity loss if the database becomes unstable.Comment: 6 pages; accepted to IEEE High Performance Extreme Computing (HPEC)
conference 2015. arXiv admin note: text overlap with arXiv:1406.492
Lustre, Hadoop, Accumulo
Data processing systems impose multiple views on data as it is processed by
the system. These views include spreadsheets, databases, matrices, and graphs.
There are a wide variety of technologies that can be used to store and process
data through these different steps. The Lustre parallel file system, the Hadoop
distributed file system, and the Accumulo database are all designed to address
the largest and the most challenging data storage problems. There have been
many ad-hoc comparisons of these technologies. This paper describes the
foundational principles of each technology, provides simple models for
assessing their capabilities, and compares the various technologies on a
hypothetical common cluster. These comparisons indicate that Lustre provides 2x
more storage capacity, is less likely to loose data during 3 simultaneous drive
failures, and provides higher bandwidth on general purpose workloads. Hadoop
can provide 4x greater read bandwidth on special purpose workloads. Accumulo
provides 10,000x lower latency on random lookups than either Lustre or Hadoop
but Accumulo's bulk bandwidth is 10x less. Significant recent work has been
done to enable mix-and-match solutions that allow Lustre, Hadoop, and Accumulo
to be combined in different ways.Comment: 6 pages; accepted to IEEE High Performance Extreme Computing
conference, Waltham, MA, 201
Lessons Learned from a Decade of Providing Interactive, On-Demand High Performance Computing to Scientists and Engineers
For decades, the use of HPC systems was limited to those in the physical
sciences who had mastered their domain in conjunction with a deep understanding
of HPC architectures and algorithms. During these same decades, consumer
computing device advances produced tablets and smartphones that allow millions
of children to interactively develop and share code projects across the globe.
As the HPC community faces the challenges associated with guiding researchers
from disciplines using high productivity interactive tools to effective use of
HPC systems, it seems appropriate to revisit the assumptions surrounding the
necessary skills required for access to large computational systems. For over a
decade, MIT Lincoln Laboratory has been supporting interactive, on-demand high
performance computing by seamlessly integrating familiar high productivity
tools to provide users with an increased number of design turns, rapid
prototyping capability, and faster time to insight. In this paper, we discuss
the lessons learned while supporting interactive, on-demand high performance
computing from the perspectives of the users and the team supporting the users
and the system. Building on these lessons, we present an overview of current
needs and the technical solutions we are building to lower the barrier to entry
for new users from the humanities, social, and biological sciences.Comment: 15 pages, 3 figures, First Workshop on Interactive High Performance
Computing (WIHPC) 2018 held in conjunction with ISC High Performance 2018 in
Frankfurt, German
Measuring the Impact of Spectre and Meltdown
The Spectre and Meltdown flaws in modern microprocessors represent a new
class of attacks that have been difficult to mitigate. The mitigations that
have been proposed have known performance impacts. The reported magnitude of
these impacts varies depending on the industry sector and expected workload
characteristics. In this paper, we measure the performance impact on several
workloads relevant to HPC systems. We show that the impact can be significant
on both synthetic and realistic workloads. We also show that the performance
penalties are difficult to avoid even in dedicated systems where security is a
lesser concern
- …